[spark] support building BTree index through procedure #6956
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull request overview
This PR introduces support for building BTree global indexes in Spark through the existing CreateGlobalIndexProcedure. BTree indexes provide efficient point lookups and range queries for high-cardinality data types like integers, doubles, and strings, complementing the existing Bitmap index implementation.
Key changes:
- Implements a custom topology builder (
BTreeIndexTopoBuilder) that uses Spark's range shuffle and sorting capabilities to distribute index building across partitions - Refactors
GlobalIndexBuilderfrom concrete class to abstract class with separate implementations for default (bitmap) and BTree indexes - Adds configuration options for controlling BTree index parallelism and records per range
- Fixes serialization bug where
extraFieldIdsnull check was incorrectly checkingindexMetafield instead
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
BTreeIndexTopoBuilder.java |
Implements distributed BTree index building using Spark range shuffle and sort; orchestrates parallel index file generation |
BTreeGlobalIndexBuilder.java |
Handles per-partition BTree index file writing with automatic flushing based on record count or partition boundaries |
BTreeGlobalIndexBuilderFactory.java |
Factory class for creating BTree index builders and topology builders via service loader pattern |
IndexFieldsExtractor.java |
Utility class for extracting partition, index field, and row ID from records during index building |
GlobalIndexTopoBuilder.java |
Interface change to support custom topology builders with direct access to SparkSession and data sources |
GlobalIndexBuilderContext.java |
Enhanced context to support nullable partition info and full range tracking for BTree indexes |
GlobalIndexBuilder.java |
Refactored to abstract class with iterator-based build method, supporting both singleton and parallel writers |
DefaultGlobalIndexBuilder.java |
Extracted default (bitmap) index building logic from the original GlobalIndexBuilder |
CreateGlobalIndexProcedure.java |
Modified to support custom topology builders that bypass traditional shard-based splitting |
BTreeIndexOptions.java |
Adds configuration options for records per range and max parallelism; fixes typo in compression-level key |
IndexManifestEntrySerializer.java |
Fixes bug where null check incorrectly evaluated indexMeta instead of extraFieldIds |
IndexFileMetaSerializer.java |
Fixes same serialization bug as IndexManifestEntrySerializer |
CreateGlobalIndexProcedureTest.scala |
Adds comprehensive tests for BTree index creation with single and multiple partitions, including overlap detection |
| Service files | Registers BTreeGlobalIndexerFactory and BTreeGlobalIndexBuilderFactory for service loader discovery |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...ark-ut/src/test/scala/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedureTest.scala
Show resolved
Hide resolved
...-common/src/main/java/org/apache/paimon/spark/globalindex/btree/BTreeGlobalIndexBuilder.java
Show resolved
Hide resolved
...-common/src/main/java/org/apache/paimon/spark/globalindex/btree/BTreeGlobalIndexBuilder.java
Show resolved
Hide resolved
...rk-common/src/main/java/org/apache/paimon/spark/globalindex/btree/BTreeIndexTopoBuilder.java
Show resolved
Hide resolved
...rk-common/src/main/java/org/apache/paimon/spark/globalindex/btree/BTreeIndexTopoBuilder.java
Outdated
Show resolved
Hide resolved
...ark-ut/src/test/scala/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedureTest.scala
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #6834
This PR is about to support creating btree index through current spark
CreateGlobalIndexProcedure.The whole process is illustrated as below:
<partition, indexed field>records-per-rangeandmax-parallelismoption.Note that the effective number of records of each btree file would not be precisely equal to
records-per-range, the reason is that: (1) spark range shuffle is implemented through sampling. (2) if a Paimon partition spans multiple Spark partitions, the first and last output files may contain relatively few records (As the green-colored index writers in the picture before).Tests
Please see
org.apache.paimon.spark.procedure.CreateGlobalIndexProcedureTestfor ut test.API and Format
This pr do not modify any existing public api.
Documentation
Will be added ASAP